{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <p class=\"display-1 h1\">语言模型和数据集</p>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 什么是语言模型？\n",
    "\n",
    "**语言模型（Language Model）**是自然语言处理中的核心任务之一，它试图对文本序列的概率分布进行建模。\n",
    "\n",
    "给定一个文本序列 $x_1, x_2, \\ldots, x_T$，语言模型的目标是估计联合概率：\n",
    "\n",
    "$$P(x_1, x_2, \\ldots, x_T)$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 为什么需要语言模型？\n",
    "\n",
    "1. **文本生成**：机器翻译、对话系统、文本摘要\n",
    "2. **文本评估**：判断文本的流畅性和合理性\n",
    "3. **语音识别**：帮助识别系统选择最可能的词序列\n",
    "4. **信息检索**：理解查询意图，提高搜索质量\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 语言模型的实际应用案例\n",
    "\n",
    "- **GPT系列模型**：GPT-3有1750亿参数，可以生成流畅的文本、代码、对话\n",
    "- **BERT模型**：双向语言模型，在多项NLP任务上取得突破\n",
    "- **机器翻译**：Google Translate、DeepL等使用序列到序列模型\n",
    "- **语音助手**：Siri、Alexa、小爱同学需要理解用户意图并生成回复\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 语言模型的挑战\n",
    "\n",
    "- **长距离依赖**：理解上下文需要记忆很远的信息\n",
    "- **组合爆炸**：可能的序列数量随长度指数增长\n",
    "- **稀疏性**：训练数据中大部分序列从未出现\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 学习语言模型\n",
    "\n",
    "### 使用序列模型\n",
    "\n",
    "根据概率的链式法则，我们可以将联合概率分解为：\n",
    "\n",
    "$$P(x_1, x_2, \\ldots, x_T) = \\prod_{t=1}^T P(x_t \\mid x_1, \\ldots, x_{t-1})$$\n",
    "\n",
    "**问题**：如何估计 $P(x_t \\mid x_1, \\ldots, x_{t-1})$？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 统计方法：n-gram模型\n",
    "\n",
    "**马尔可夫（Markov）假设**：一个词的出现只依赖于前面 $n-1$ 个词\n",
    "\n",
    "- **一元语法（unigram）**：$P(x_t)$ - 词独立出现\n",
    "- **二元语法（bigram）**：$P(x_t \\mid x_{t-1})$ - 只依赖前一个词\n",
    "- **三元语法（trigram）**：$P(x_t \\mid x_{t-1}, x_{t-2})$ - 依赖前两个词\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### n-gram模型的数学形式\n",
    "\n",
    "对于n-gram模型，条件概率通过最大似然估计（MLE）计算：\n",
    "\n",
    "$$P(x_t \\mid x_{t-n+1}, \\ldots, x_{t-1}) = \\frac{C(x_{t-n+1}, \\ldots, x_t)}{C(x_{t-n+1}, \\ldots, x_{t-1})}$$\n",
    "\n",
    "其中$C(\\cdot)$表示计数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 平滑技术（Smoothing）\n",
    "\n",
    "\n",
    "n-gram模型面临：**零概率问题**（Zero Probability Problem）\n",
    "\n",
    "**示例**：\n",
    "假设训练数据中从未出现过\"the cat sat\"，那么：\n",
    "$$P(\\text{sat} \\mid \\text{the cat}) = \\frac{C(\\text{the cat sat})}{C(\\text{the cat})} = \\frac{0}{C(\\text{the cat})} = 0$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**后果**：\n",
    "- 如果测试数据中出现未见的n-gram，整个序列的概率变为0\n",
    "- 即使其他部分很合理，整个句子也会被判为\"不可能\"\n",
    "- 模型无法处理训练数据中未出现的组合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**平滑技术的作用**：\n",
    "- 给未见的n-gram分配一个小的非零概率\n",
    "- 避免零概率问题\n",
    "- 提高模型的泛化能力"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 拉普拉斯平滑（Laplace Smoothing）\n",
    "\n",
    "**核心思想**：\n",
    "- 给每个n-gram的计数加1（或加$k$）\n",
    "- 同时调整分母，确保概率和为1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**数学形式**：\n",
    "\n",
    "**原始MLE公式**：\n",
    "$$P(x_t \\mid x_{t-n+1}, \\ldots, x_{t-1}) = \\frac{C(x_{t-n+1}, \\ldots, x_t)}{C(x_{t-n+1}, \\ldots, x_{t-1})}$$\n",
    "\n",
    "**拉普拉斯平滑公式**（加1平滑）：\n",
    "$$P(x_t \\mid x_{t-n+1}, \\ldots, x_{t-1}) = \\frac{C(x_{t-n+1}, \\ldots, x_t) + 1}{C(x_{t-n+1}, \\ldots, x_{t-1}) + |V|}$$\n",
    "\n",
    "其中$|V|$是词汇表大小。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 拉普拉斯平滑示例\n",
    "\n",
    "**假设**：\n",
    "- 词汇表大小：$|V| = 1000$\n",
    "- 训练数据中：\"the cat\"出现10次\n",
    "- \"the cat sat\"出现3次\n",
    "- \"the cat ran\"出现2次\n",
    "- \"the cat ate\"出现5次"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**原始MLE**：\n",
    "- $P(\\text{sat} \\mid \\text{the cat}) = \\frac{3}{10} = 0.3$\n",
    "- $P(\\text{ran} \\mid \\text{the cat}) = \\frac{2}{10} = 0.2$\n",
    "- $P(\\text{ate} \\mid \\text{the cat}) = \\frac{5}{10} = 0.5$\n",
    "- $P(\\text{jumped} \\mid \\text{the cat}) = \\frac{0}{10} = 0$ ❌ **零概率问题！**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**拉普拉斯平滑后**：\n",
    "- $P(\\text{sat} \\mid \\text{the cat}) = \\frac{3+1}{10+1000} = \\frac{4}{1010} \\approx 0.004$\n",
    "- $P(\\text{ran} \\mid \\text{the cat}) = \\frac{2+1}{10+1000} = \\frac{3}{1010} \\approx 0.003$\n",
    "- $P(\\text{ate} \\mid \\text{the cat}) = \\frac{5+1}{10+1000} = \\frac{6}{1010} \\approx 0.006$\n",
    "- $P(\\text{jumped} \\mid \\text{the cat}) = \\frac{0+1}{10+1000} = \\frac{1}{1010} \\approx 0.001$ ✅ **非零概率！**\n",
    "\n",
    "**观察**：\n",
    "- 未见的\"jumped\"现在有非零概率\n",
    "- 但概率很小，符合\"少见\"的直觉\n",
    "- 常见组合的概率被\"稀释\"，但仍然相对较高"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 拉普拉斯平滑的优缺点\n",
    "\n",
    "**优点**：\n",
    "- **解决零概率问题**：未见的n-gram也有概率\n",
    "- **简单易实现**：只需修改计数公式\n",
    "- **理论基础**：基于贝叶斯估计，有理论保证\n",
    "- **保证概率归一化**：所有概率之和为1\n",
    "\n",
    "**缺点**：\n",
    "- **过度平滑**：对于大词汇表，概率被\"稀释\"得太厉害\n",
    "- **不区分罕见词和常见词**：所有未见的n-gram概率相同\n",
    "- **对高频n-gram不公平**：常见组合的概率被降低\n",
    "\n",
    "**改进方法**：\n",
    "- **加$k$平滑**：不是加1，而是加一个小的常数$k$（如0.5）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 其他平滑技术\n",
    "\n",
    "**回退（Backoff）**：\n",
    "- 当n-gram不存在时，使用(n-1)-gram\n",
    "- 例如：如果\"the cat sat\"不存在，使用$P(\\text{sat} \\mid \\text{cat})$\n",
    "- 如果\"cat sat\"也不存在，使用$P(\\text{sat})$\n",
    "\n",
    "**插值（Interpolation）**：\n",
    "- 组合不同阶的n-gram，加权平均\n",
    "- 例如：$P(\\text{sat} \\mid \\text{the cat}) = \\lambda_1 P(\\text{sat} \\mid \\text{the cat}) + \\lambda_2 P(\\text{sat} \\mid \\text{cat}) + \\lambda_3 P(\\text{sat})$\n",
    "- 其中$\\lambda_1 + \\lambda_2 + \\lambda_3 = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### n-gram模型的局限性\n",
    "\n",
    "- 需要存储所有可能的n-gram组合\n",
    "- 无法捕获长距离依赖\n",
    "- 数据稀疏问题严重\n",
    "- 即使使用平滑，仍然无法很好地处理未见的序列组合\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 神经网络方法\n",
    "\n",
    "使用神经网络（特别是RNN、LSTM、Transformer）可以：\n",
    "\n",
    "- **自动学习特征**：不需要手工设计n-gram\n",
    "- **捕获长距离依赖**：通过隐藏状态传递信息\n",
    "- **泛化能力**：相似上下文共享参数\n",
    "\n",
    "**核心思想**：将词序列映射到固定维度的向量表示，然后使用神经网络建模条件概率。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 困惑度（Perplexity）\n",
    "\n",
    "**定义**：\n",
    "\n",
    "$$PP = \\exp\\left(-\\frac{1}{T}\\sum_{t=1}^T \\log P(x_t \\mid x_1, \\ldots, x_{t-1})\\right)$$\n",
    "\n",
    "**含义**：\n",
    "- 困惑度越低，模型越好\n",
    "- 可以理解为\"平均分支因子\"（平均有多少个可能的下一个词）\n",
    "- 完美模型的困惑度为1（每一步都以概率1准确预测出下一个词）\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 自然语言统计\n",
    "\n",
    "### 数据集介绍\n",
    "\n",
    "使用《时光机器》这本小说作为数据集，演示语言模型的构建过程。\n",
    "\n",
    "**数据集特点**：\n",
    "- 英文文本，适合演示n-gram模型\n",
    "- 文本长度适中，便于分析\n",
    "- 包含丰富的词汇和语法结构\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 预处理步骤\n",
    "\n",
    "1. 读取文本文件\n",
    "2. 分词（tokenization）：将文本分割成词元\n",
    "3. 构建词汇表：统计词频，建立映射关系\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:07.483736Z",
     "iopub.status.busy": "2022-07-31T02:26:07.483322Z",
     "iopub.status.idle": "2022-07-31T02:26:07.522052Z",
     "shell.execute_reply": "2022-07-31T02:26:07.521400Z"
    },
    "origin_pos": 4,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 2261),\n",
       " ('i', 1267),\n",
       " ('and', 1245),\n",
       " ('of', 1155),\n",
       " ('a', 816),\n",
       " ('to', 695),\n",
       " ('was', 552),\n",
       " ('in', 541),\n",
       " ('that', 443),\n",
       " ('my', 440)]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "import torch\n",
    "from d2l import torch as d2l\n",
    "\n",
    "tokens = d2l.tokenize(d2l.read_time_machine())\n",
    "corpus = [token for line in tokens for token in line]\n",
    "vocab = d2l.Vocab(corpus)\n",
    "vocab.token_freqs[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Zipf定律和停用词\n",
    "\n",
    "**Zipf定律**：在自然语言文本中，词频 $f$ 与排名 $r$ 的关系为：\n",
    "\n",
    "$$f \\propto \\frac{1}{r^\\alpha}$$\n",
    "\n",
    "其中 $\\alpha \\approx 1$。\n",
    "\n",
    "**含义**：\n",
    "- 少数高频词（停用词）占据了大部分文本\n",
    "- 大量低频词（长尾分布）构成了词汇表的主体\n",
    "- 这对语言模型的训练和评估提出了挑战\n",
    "\n",
    "**停用词（Stop Words）**：\n",
    "最流行的词（如\"the\", \"a\", \"of\"）被称为**停用词**，它们：\n",
    "- 出现频率极高但信息量低\n",
    "- 在文本预处理中常被过滤\n",
    "- 但在语言模型中仍需建模（因为它们是语法的重要组成部分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 对数坐标的优势\n",
    "\n",
    "- 可以同时显示高频词和低频词\n",
    "- 更容易观察幂律分布（power law）\n",
    "- 便于比较不同n-gram模型的分布差异\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 词频分布可视化\n",
    "\n",
    "使用对数坐标绘制词频分布图，可以清晰地看到Zipf定律的体现：\n",
    "\n",
    "**对数坐标的优势**：\n",
    "- 可以同时显示高频词和低频词\n",
    "- 更容易观察幂律分布（power law）\n",
    "- 便于比较不同n-gram模型的分布差异\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 二元语法（Bigram）\n",
    "\n",
    "- 考虑相邻两个词的组合\n",
    "- 使用`zip`函数将相邻的词配对：`corpus[:-1]`和`corpus[1:]`创建滑动窗口\n",
    "- 二元组合数量 = 词汇表大小²（理论上）\n",
    "- 实际出现的组合远少于理论值（数据稀疏）\n",
    "- 常见组合多为固定搭配（如\"of the\", \"in the\"）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 三元语法（Trigram）\n",
    "\n",
    "- 考虑连续三个词的组合\n",
    "- 使用三个偏移的序列创建三元组合：`corpus[:-2]`, `corpus[1:-1]`, `corpus[2:]`形成滑动窗口\n",
    "- 三元组合数量 = 词汇表大小³（理论上）\n",
    "- 数据稀疏问题更加严重\n",
    "- 但能捕获更多语法和语义信息\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 复杂度分析\n",
    "\n",
    "- 时间复杂度：O(n)，其中n是语料库长度\n",
    "- 空间复杂度：O(V³)，其中V是词汇表大小\n",
    "- 实际存储的组合数远小于V³（数据稀疏）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### n-gram模型的观察与启示\n",
    "\n",
    "- 随着n增大，组合数量指数增长\n",
    "- 长尾分布更加明显\n",
    "- n越大，模型表达能力越强，但数据稀疏问题越严重\n",
    "- 需要在模型复杂度和数据需求之间找到平衡\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 序列数据的挑战\n",
    "\n",
    "在训练语言模型时，我们需要处理长文本序列。主要挑战：\n",
    "\n",
    "1. **序列长度不一致**：不同文本长度差异很大\n",
    "2. **计算效率**：不能一次性处理整个序列\n",
    "3. **内存限制**：需要分批处理数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 序列数据的特殊性质\n",
    "\n",
    "**与图像数据的区别**：\n",
    "- **顺序性**：序列中元素的位置很重要\n",
    "- **变长性**：不同序列长度可能不同\n",
    "- **依赖关系**：前后元素之间存在依赖\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 处理策略\n",
    "\n",
    "- **填充（Padding）**：将短序列填充到相同长度\n",
    "- **截断（Truncation）**：将长序列截断到固定长度\n",
    "- **动态批处理**：将相似长度的序列组织在一起\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 两种采样策略\n",
    "\n",
    "#### 1. 随机采样（Random Sampling）\n",
    "\n",
    "**特点**：每个样本是原始序列上任意捕获的子序列\n",
    "\n",
    "**示例说明**（假设原始序列为0-34，batch_size=2，num_steps=5）：\n",
    "\n",
    "**第一个批次**：\n",
    "```\n",
    "X: tensor([[14, 15, 16, 17, 18],    # 第一个样本（第0行）\n",
    "           [ 9, 10, 11, 12, 13]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第二个批次**：\n",
    "```\n",
    "X: tensor([[19, 20, 21, 22, 23],    # 第一个样本（第0行）\n",
    "           [29, 30, 31, 32, 33]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第三个批次**：\n",
    "```\n",
    "X: tensor([[ 4,  5,  6,  7,  8],    # 第一个样本（第0行）\n",
    "           [24, 25, 26, 27, 28]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**关键观察**：\n",
    "- **每个样本（每行）内部是连续的**：第一个样本 14→15→16→17→18，第二个样本 9→10→11→12→13\n",
    "- **批次之间没有顺序关系**：第一个批次是14-18和9-13，第二个批次是19-23和29-33，第三个批次是4-8和24-28（顺序被打乱）\n",
    "- **样本顺序被打乱**：原始序列的顺序被破坏，样本之间相互独立\n",
    "\n",
    "这种设计使样本之间独立，有利于训练稳定，但破坏了序列的连续性\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 随机采样的优点\n",
    "\n",
    "- 样本之间独立，训练稳定\n",
    "- 适合训练初期，快速探索数据分布\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 随机采样的缺点\n",
    "\n",
    "- 破坏了序列的连续性，不利于学习长距离依赖\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 2. 顺序采样（Sequential Partitioning）\n",
    "\n",
    "**特点**：相邻小批量中的子序列在原始序列上也是相邻的\n",
    "\n",
    "**示例说明**（假设原始序列为0-34，batch_size=2，num_steps=5）：\n",
    "\n",
    "**第一个批次**：\n",
    "```\n",
    "X: tensor([[ 4,  5,  6,  7,  8],    # 第一个样本（第0行）\n",
    "           [19, 20, 21, 22, 23]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第二个批次**：\n",
    "```\n",
    "X: tensor([[ 9, 10, 11, 12, 13],    # 第一个样本（第0行）\n",
    "           [24, 25, 26, 27, 28]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**关键观察**：\n",
    "- **每个样本（每行）内部是连续的**：第一个样本 4→5→6→7→8，第二个样本 19→20→21→22→23\n",
    "- **同一批次内不同样本（不同行）之间不连续**：第一个样本从位置4开始，第二个样本从位置19开始（它们是并行序列）\n",
    "- **相邻批次之间每个样本（每行）是连续的**：第一个样本 4→9→14（跨批次连续），第二个样本 19→24→29（跨批次连续）\n",
    "\n",
    "这种设计允许并行处理多个序列（提高效率），同时保持每个序列的连续性（有利于RNN学习）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### 顺序采样的优点\n",
    "\n",
    "- 保持序列连续性，有利于学习序列结构\n",
    "- 在RNN训练中，隐藏状态可以在相邻批次间传递\n",
    "- 更好地学习序列的长期依赖关系\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### 顺序采样的缺点\n",
    "\n",
    "- 样本之间不独立，可能导致梯度估计有偏\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 采样策略的选择建议\n",
    "\n",
    "**何时使用随机采样**：\n",
    "- 训练初期：快速探索数据分布\n",
    "- 数据量很大：需要打乱数据顺序\n",
    "- 模型较简单：不需要长距离依赖\n",
    "\n",
    "**何时使用顺序采样**：\n",
    "- RNN/LSTM训练：需要保持序列连续性\n",
    "- 长文本建模：需要捕获长期依赖\n",
    "- 序列到序列任务：编码器需要完整上下文\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 混合策略\n",
    "\n",
    "- 训练初期使用随机采样\n",
    "- 训练后期切换到顺序采样\n",
    "- 或者交替使用两种策略\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 随机采样关键步骤\n",
    "\n",
    "1. **随机起始位置**：随机丢弃前几个词元，增加随机性\n",
    "2. **计算子序列数量**：确保每个子序列都有对应的标签\n",
    "3. **打乱顺序**：破坏序列顺序，使样本独立\n",
    "4. **生成批次**：使用生成器模式，每次yield一个批次的数据\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 随机采样函数实现\n",
    "\n",
    "**代码实现解析**：\n",
    "\n",
    "1. **随机起始位置**：\n",
    "   ```python\n",
    "   corpus = corpus[random.randint(0, num_steps - 1):]\n",
    "   ```\n",
    "   随机丢弃前几个词元，增加随机性\n",
    "\n",
    "2. **计算子序列数量**：\n",
    "   ```python\n",
    "   num_subseqs = (len(corpus) - 1) // num_steps\n",
    "   ```\n",
    "   确保每个子序列都有对应的标签\n",
    "\n",
    "3. **打乱顺序**：\n",
    "   ```python\n",
    "   random.shuffle(initial_indices)\n",
    "   ```\n",
    "   破坏序列顺序，使样本独立\n",
    "\n",
    "4. **生成批次**：\n",
    "   - 使用生成器模式，节省内存\n",
    "   - 每次yield一个批次的数据\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 顺序采样关键步骤\n",
    "\n",
    "1. **随机偏移**：增加随机性，避免总是从序列开头开始\n",
    "2. **对齐到批次大小**：确保可以均匀分割成batch_size个并行序列\n",
    "3. **重塑为并行序列**：将一维序列重塑为二维矩阵，每行是一个并行序列\n",
    "4. **顺序采样**：每个并行序列内部按顺序采样，相邻批次之间保持连续性\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 顺序采样函数实现\n",
    "\n",
    "**代码实现解析**：\n",
    "\n",
    "1. **随机偏移**：\n",
    "   ```python\n",
    "   offset = random.randint(0, num_steps)\n",
    "   ```\n",
    "   增加随机性，避免总是从序列开头开始\n",
    "\n",
    "2. **对齐到批次大小**：\n",
    "   ```python\n",
    "   num_tokens = ((len(corpus) - offset - 1) // batch_size) * batch_size\n",
    "   ```\n",
    "   确保可以均匀分割成batch_size个并行序列\n",
    "\n",
    "3. **重塑为并行序列**：\n",
    "   ```python\n",
    "   Xs, Ys = Xs.reshape(batch_size, -1), Ys.reshape(batch_size, -1)\n",
    "   ```\n",
    "   将一维序列重塑为二维矩阵，每行是一个并行序列\n",
    "\n",
    "4. **顺序采样**：\n",
    "   - 每个并行序列内部按顺序采样\n",
    "   - 相邻批次之间保持连续性\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 策略模式（Strategy Pattern）\n",
    "\n",
    "- 将不同的采样算法封装成独立的策略\n",
    "- 通过`use_random_iter`参数运行时动态选择策略\n",
    "- 符合开闭原则（对扩展开放，对修改关闭）\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 迭代器模式（Iterator Pattern）\n",
    "\n",
    "- 实现`__iter__`方法，返回迭代器\n",
    "- 支持`for`循环遍历\n",
    "- 延迟计算，节省内存\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 工厂模式（Factory Pattern）\n",
    "\n",
    "- `load_data_time_machine`函数封装对象创建过程\n",
    "- 简化客户端代码，提供统一接口\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 函数参数说明\n",
    "\n",
    "- `batch_size`：批次大小\n",
    "- `num_steps`：序列长度（时间步数）\n",
    "- `use_random_iter`：是否使用随机采样（默认False，使用顺序采样）\n",
    "- `max_tokens`：限制词汇表大小，过滤低频词\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 函数返回值\n",
    "\n",
    "- `data_iter`：数据迭代器，用于训练循环\n",
    "- `vocab`：词汇表，包含词到索引的映射\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 数据加载器设计\n",
    "\n",
    "`SeqDataLoader`类封装了数据加载逻辑，采用多种设计模式：\n",
    "\n",
    "**策略模式（Strategy Pattern）**：\n",
    "- 将不同的采样算法封装成独立的策略\n",
    "- 通过`use_random_iter`参数运行时动态选择策略\n",
    "- 符合开闭原则（对扩展开放，对修改关闭）\n",
    "\n",
    "**迭代器模式（Iterator Pattern）**：\n",
    "- 实现`__iter__`方法，返回迭代器\n",
    "- 支持`for`循环遍历\n",
    "- 延迟计算，节省内存\n",
    "\n",
    "**接口设计**：\n",
    "- `__init__`：初始化参数，加载数据\n",
    "- `__iter__`：返回数据迭代器\n",
    "- 符合Python迭代器协议，易于使用\n",
    "\n",
    "**工厂模式（Factory Pattern）**：\n",
    "- `load_data_time_machine`函数封装对象创建过程\n",
    "- 简化客户端代码，提供统一接口\n",
    "\n",
    "**返回值**：\n",
    "- `data_iter`：数据迭代器，用于训练循环\n",
    "- `vocab`：词汇表，包含词到索引的映射\n",
    "\n",
    "**参数说明**：\n",
    "- `batch_size`：批次大小\n",
    "- `num_steps`：序列长度（时间步数）\n",
    "- `use_random_iter`：是否使用随机采样（默认False，使用顺序采样）\n",
    "- `max_tokens`：限制词汇表大小，过滤低频词\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 总结与思考\n",
    "\n",
    "### 关键概念回顾\n",
    "\n",
    "1. **语言模型**：估计文本序列的概率分布\n",
    "2. **n-gram模型**：基于马尔可夫假设的统计方法\n",
    "3. **Zipf定律**：自然语言中的词频分布规律\n",
    "4. **采样策略**：随机采样 vs 顺序采样\n",
    "\n",
    "### 思考问题\n",
    "\n",
    "1. 为什么n-gram模型在n较大时会出现数据稀疏问题？\n",
    "2. 随机采样和顺序采样各适用于什么场景？\n",
    "3. 如何平衡模型复杂度和数据需求？\n",
    "4. 现代神经网络语言模型如何解决n-gram模型的局限性？\n",
    "\n",
    "### 下一步学习\n",
    "\n",
    "- RNN架构：如何用循环神经网络建模序列\n",
    "- 长距离依赖：LSTM和GRU如何解决梯度消失问题\n",
    "- 现代方法：Transformer和注意力机制\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 三种模型的对比可视化\n",
    "\n",
    "通过对比一元、二元、三元语法的词频分布，我们可以直观地看到：\n",
    "1. 随着n增大，组合数量指数增长\n",
    "2. 长尾分布更加明显\n",
    "3. 数据稀疏问题加剧\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\frank\\venvTorch\\lib\\site-packages\\ipykernel_launcher.py:5: RuntimeWarning: invalid value encountered in log\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1e27ae37cf8>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAb70lEQVR4nO3deXRddb338fc3SdM285y0GZqEjulc0tKKDFKGeikCXh8FHLioF3U9irPXCV3eweV9rsvhWVyvTxEvOIGAIMpFBgUtAp0pHZMOSdqm7cnYzD2Zzu/544RSGdvm5Oyzz/m81mI1OQn7fPYi+fDrb/9+e5tzDhER8a8krwOIiMj4qMhFRHxORS4i4nMqchERn1ORi4j4XIoXb1pQUOAqKyu9eGsREd/aunVru3Ou8NWve1LklZWVbNmyxYu3FhHxLTM79Hqva2pFRMTnVOQiIj6nIhcR8TkVuYiIz6nIRUR8TkUuIuJzKnIREZ/zZB25iEgicM7R2T9EU0c/je0DHOro57215ZTnpUX0fVTkIiLj4JzjxMAwje39HOrop6m9n8aOAZra+2nq6Kc3OHLqe5OTjGUVuSpyEREv9ASHaWjrp7G9j8b2cFEf6uinsb2fntPKOsmgNHcqlfnpXF9RSmV+OpUFaVTmp1OWm0ZqSuRntFXkIiJjRkZDNJ84SUN7Hw1t/Rxs6+NgWz8Nbf209w2e+j4zKM2ZSlVBOtcuKWVGfhpVBelUFqRTPkFl/WZU5CKScLoGhsYKuo+G9vCfB9vCI+zh0Vcef5mbNonqwgwum1tIdWEG1QXpVBemU56XxuSUZA/P4G+pyEUkLg2PhjjSOXBqZN3Q1n9qpN3RP3Tq+yYlGxV5aVQXZnD5vGKqC9M5rzCd6oIMctNTPTyDM6ciFxFfGxwZpbG9n/0tfexv7eNAay/7W/pobO9nJPTK6LogI5XqggyuqHm5rDOoLsygPHcqKcn+XomtIhcRXwgOj9LQ1s/+saLe39rL/tY+DnUMMDpW2EkGM/LTmVmUweU1xWNlnc55BRlkp03y+AwmjopcRGLKyaFRDraFi3pfSx/7W8Kj7MOdA7w8wE5OMmbkpzG7KJOrF05jZlEGs4oyqS5MZ8qk2Jm7jhYVuYh4YmQ0RFPHAPWBXuoCPdQFeqkP9HLkxABurLBTkoyqgnRqpmfxriWlzC4OF3ZlQWxdbPSailxEJlxb7+DfFHZdoIf9LX0MjoSA8JRIVUE6C0uz+ftlZcwqzmBWUQaVBelM8vn8dTSMu8jNrBz4GVAMOGCdc+6H4z2uiPhPcHiU/S197A30nCru+kAv7X2vrBIpyJjMvGmZfHDlDOZOy2JuSSYzizISckokUiIxIh8BPu+c22ZmmcBWM3vKObcnAscWkRjknKOlZ5Ddx7rZc6yHvWMj7ab2/lPz2FMmJTG7OJN3zCk6VdhzSjIpyJjsbfg4NO4id84dB46PfdxrZnuBUkBFLhIHQiFHU0c/u4/1jP0TLu/T12LPyE9jbkkmaxdNZ95YYc/ITyc5yTxMnjgiOkduZpXAUmDj63ztVuBWgIqKiki+rYhEyOBIeGpkz1hh7z7Ww97jPfQPjQLhzTOzijK5bG4R86dnUTM9m3nTMsmcEr9L+/wgYkVuZhnAb4DPOOd6Xv1159w6YB1AbW2te/XXRSS6gsOj1AV62dHcxc7mbnYd6+FAa++pLerpqcnMm5bFe84vY/70bGqmZzGrOEOrRWJQRIrczCYRLvFfOuceisQxRSRyhkdD7G/pY0dzFzuOdrOjuYv6wCulnZ+eyvzSbC6dU0jNtCzmT8+iMj+dJE2N+EIkVq0YcBew1zn3vfFHEpHxCIUcDe394dJuDpf27mM9p5b6ZU5JYVFZNh+9qJpFpdksKs9hevYUwr/K4keRGJFfCHwQ2Glm28de+6pz7rEIHFtE3sLx7pNsO9TFS81d7GjuYtfRHvoGw/fHnjopmQWlWXxg5QwWlWWzqCyHGXlpGmnHmUisWvkroJ8KkSgIDo+y+1gPLx4+wbbDJ9h2qItATxCA1OQk5k3L5PqlpSwsy2ZxWQ4zizK0ciQBaGenSIxyznGsOxgu7UNdbDt8gj3HehgaDU+RlOZMZXlVHssqclhWkcu8aVlRf6CBxAYVuUiMCI+2u0+V9rbDJ2jpCT+VZnJKEovLcrjlwkqWVuSyrCKHoqwpHieWWKEiF/FI98lhth7qZGNjJ5sbO9l5tPvUKpLyvKmsrM5nWUUuSytymDctS/cckTekIheJkpaeIJsaO9nc1Mmmxk7qW3pxLrzJZmFpNh9+exXnV+SytCKXwkxtY5czpyIXmQDOOZo6Btjc2MmmpnB5H+oYACAtNZnzZ+Tydwunsbwyj6UVObphlIyLilwkQo50DvDCwQ6eP9jO8wc7aO0Nz2/npadSOyOXD66cwYqqPGqmZfn+0WISW1TkIuco0B3khYb2sfLuoPnESSB8m9a3nZfPBdV5XFCVx3mFGdpsIxNKRS5yhjr6BtnQ0MnzB9t5oaGDhrZ+ALKnTmJVdT63XlzNqup8ZhapuCW6VOQibyA4PMrmpk6e3d/O+n1t1AV6gfDNpC6ozuemFRWsrM6nZlqWdkqKp1TkImOccxxs6+Mv+8LFvbGxg+BwiNTkJGorc/niVXNYdV4+C0uztRRQYoqKXBJa98Awzx0MF/f6fW0c6w5vd68uSOeG5RVcPLuAldX5pKXqV0Vil346JaGEQo4dR7v5c30r6/e1sf1IFyEHmZNTuHBmAZ+8rJCLZhVQnpfmdVSRM6Yil7jXPzjCs/vbebquhafr2mjvG8QMFpXl8Ml3zOTi2YUsKc/RkkDxLRW5xKUjnQM8XdfKn+pa2XCwg6HREJlTUrh0ThGr5xZx8exC8tJTvY4pEhEqcokLzjl2NHfzxO4Af9rbSn1LeIVJdUE6N79tBpfNLaa2MlcXKSUuqcjFt0ZGQ2xuOsETuwM8uTvAse4gyUnGiso8vn71PC6bW0R1YYbXMUUmnIpcfCU4PMpzB9p5YneAp/a0cGJgmMkpSVw8u5DPXzmH1fOKyEnTlIkkFhW5xLzg8Ch/rm/l0R3Heaaulf6hUTInp7B6XhFXzS/hkjmFWh4oCU0//RKThkZCPLu/jUd3HOfJ3QH6h0YpyEjlXUtKWbOghFXV+XoajsgYFbnEjJHREC80dPD7l47x+K4APcERsqdO4prF07lm8XQuqMrTEkGR16EiF08559hy6ASPbD/KH3YG6OgfImNyClfWFHPN4ulcOLNAI2+Rt6AiF08c6RzgN9uaeWjbUQ53DjBlUhKXzytm7aLpXDqnUA9aEDkLKnKJmt7gMH/YGeDBbc1sauzEDFZV5/Pp1bNYs6CE9Mn6cRQ5F/rNkQk1GnI8d6Cd32xr5ondAYLDIaoL0vniVXO4bmkppTlTvY4o4nsqcpkQx7tPct+mI9y/5QjHu4NkT53Ee84v493LylhanqMHL4hEkIpcImY05PjLvlZ+tfEwT9e14oCLZhVy+9oaVs8rYnKK5r1FJoKKXMYt0B3k15uP8OvNhznWHaQwczKfuPQ8blheodvBikSBilzOSSjk+Mv+tlOj79GQ46JZBdy+tobLa4p1cyqRKFKRy1npDQ7z4NZm7nm+iaaOAQoyUrn14mpuXF5BRb5G3yJeUJHLGTnU0c/dzzfxwJZm+gZHWFaRw+eunMOa+SXasCPiMRW5vCHnHBsbO7lzfQNP17eSbMbaRdO45cIqFpfneB1PRMaoyOU1RkOOJ3cH+PH6Bl460kV+eiqfesdM3r9yBsVZU7yOJyKvoiKXU4LDozy07Sh3PttAY3s/M/LT+NfrFvCe88u0ZV4khqnIhb7BEX7+wiHu+msj7X2DLCrL5kfvX8ZV80tITtLGHZFYF5EiN7OfAmuBVufcgkgcUyZeb3CYn71wiDufbaBrYJiLZhXwiUuXsKo6XzsvRXwkUiPyu4E7gJ9F6HgygXqCw9z9XBN3/bWR7pPDXDa3iNtWz2KJLmCK+FJEitw5t97MKiNxLJk4fYMj3PVsIz/5awO9wREunxcu8EVlOV5HE5FxiNocuZndCtwKUFFREa23FWBwZJRfbTzMHU8foKN/iCtqivn06lksKM32OpqIREDUitw5tw5YB1BbW+ui9b6JbDTkeGT7Ub731D6aT5xkVXU+X1ozh6UVuV5HE5EI0qqVOOSc45n6Vv7P4/XUBXqZPz2Lb1+/kItmFegipkgcUpHHmX0tvfzLo3t4dn87lflp3HHTUv5uwTSStIxQJG5FavnhvcClQIGZNQPfdM7dFYljy5npGhji+0/t4xcbD5Oemsw3r6nhAytn6C6EIgkgUqtWbozEceTsjYyG+NWmw3zvqX30nBzm/RfM4LNXzCYvPdXraCISJZpa8bFNjZ3c/ttd1Lf0sqo6n2++q4a5JVlexxKRKFOR+1D3wDDfeXwv9246QmnOVH78gfO5an6xLmSKJCgVuY8453h0x3G+9fs9nBgY4h8vquKzV8wmLVX/GUUSmRrAJ450DnD7I7v4c30bC0uzufuW5drQIyKAijzmhUKOn73QxL8/Xo8ZfGNtDTe/rVJ3JRSRU1TkMex490m++MAO/nqgnUvnFPJv1y+kNGeq17FEJMaoyGPUI9uPcvtvdzE86vj29Qu5cUW5LmaKyOtSkceYroEhbn9kN79/6RhLK3L4/nuXUFmQ7nUsEYlhKvIY8tyBdj5//0u09w3yhStn8/FLziNFOzNF5C2oyGNAKOS445kDfP+P+6guSOfOD13IwjKtSBGRM6Mi99iJ/iE+e/92/lzfxnVLpvPtdy/UunAROStqDA/taO7iE7/YRlvvIP9y3QI+cEGFLmiKyFlTkXvAOccvNx7mn3+/h8LMyTzw8VUs1vMyReQcqcijLDg8ylcf3slD245yyexCfvC+JeTqToUiMg4q8ihq6x3kYz/fwrbDXXzm8lncdtksPfBBRMZNRR4le4/38NF7ttDRP8h/vX8Z71w4zetIIhInVORR8Oz+Nj7+861kTEnhgY+9TUsLRSSiVOQT7LcvHuULD7zEzKIM7r5lBSXZU7yOJCJxRkU+gdatP8i3H6tjZXUe6z5US9aUSV5HEpE4pCKfAM45/uOJen7054NcvWga33vvYianJHsdS0TilIo8wpxzfOv3e7j7+SZuuqCCf712gVamiMiEUpFH0GjI8bWHd3Lf5iN85O1VfP3qedqpKSITTkUeIaGQ4ysP7eD+Lc186rKZfO6K2SpxEYkKFXkEOOf4xu92cf+WZm67bCafu3KO15FEJIHoZtfj5Jzjnx/dwy82HObjl5zHZ6+Y7XUkEUkwKvJx+v5T+/jv55r48IVV/NOaOZpOEZGoU5GPw883HOL/Pn2A99aWcftaXdgUEW+oyM/R47uO841HdrF6bhHfvn6hSlxEPKMiPwebmzq57b7tLC3P4Y6blum5miLiKTXQWTrSOcDHfr6Vspyp3HXzcqamasemiHhLRX4W+gZH+Og9WxgZDfGTm2v1QAgRiQlaR36GRkOOT9/7Igfa+rjnlhVUF2Z4HUlEBNCI/Ix9/6l9/KmulW9eU8PbZxV4HUdE5BQV+Rl4pr6VO545wPtqy/nQqkqv44iI/I2IFLmZrTGzejM7YGZfjsQxY8WxrpN87tfbmVuSybeune91HBGR1xh3kZtZMvCfwDuBGuBGM6sZ73FjwfBoiE/+ahtDIyF+9P5lTJmkFSoiEnsiMSJfARxwzjU454aA+4BrI3Bcz333yXq2He7iO3+/SBc3RSRmRaLIS4Ejp33ePPba3zCzW81si5ltaWtri8DbTqxNjZ2sW9/AjSsquGbxdK/jiIi8oahd7HTOrXPO1TrnagsLC6P1tuekb3CEzz+wnfLcNL5+9Tyv44iIvKlIrCM/CpSf9nnZ2Gu+9W//s5fmEye5/2OrSJ+spfYiEtsiMSLfDMwysyozSwVuAH4XgeN64pm6Vu7ddJhbL6pmeWWe13FERN7SuIebzrkRM/sk8ASQDPzUObd73Mk80D84wlcf3sns4gw9IEJEfCMi8wbOuceAxyJxLC/94I/7ON4d5I6bVmmpoYj4hnZ2jtl7vIefPtfEDcvLOX+GplRExD9U5EAo5Pj6b3eRPXUS/7RmrtdxRETOiooceHBrM1sPneDL75yrW9OKiO8kfJH3Bof598frWF6Zy3uWlXkdR0TkrCV8kd+5voGO/iFuX1tDUpKeuyki/pPQRd7aG+TOZxtZu2gai8pyvI4jInJOErrIf/jH/QyPhvjClXO8jiIics4Stsgb2vq4b/MRbrqggsqCdK/jiIics4Qt8u8+Wc+UlCRuWz3L6ygiIuOSkEVeH+jlsZ0BPvL2KgoyJnsdR0RkXBKyyP/fXw6SlprMLRdWeR1FRGTcEq7Im08M8MhLx7hxRYU2/4hIXEi4Iv/Js40Y8JG3azQuIvEhoYq8o2+Q+zYf5rqlpUzPmep1HBGRiEioIr/nhUMEh0N8/JJqr6OIiERMwhT5yaFR7nm+iStriplZlOl1HBGRiEmYIn90xzG6Tw7zYc2Ni0icSZgiv2/zEaoL07mgSg+NEJH4khBFvq+ll62HTnDj8grMdIdDEYkvCVHk9246zKRk493LSr2OIiIScXFf5MHhUR5+8ShXzS8hX9vxRSQOxX2RP7E7QNfAMDeuqPA6iojIhIj7Iv/VxsPMyE9jVXW+11FERCZEXBf5oY5+NjZ28r7l5XqMm4jErbgu8sd2BgC4dokucopI/IrrIn98d4DFZdmU6r4qIhLH4rbIj3Wd5KUjXaxZMM3rKCIiEypui/zxXeFplTULSjxOIiIyseK6yOeWZFKlByuLSJyLyyJv7Q2y+VCnRuMikhDissif3N2Cc/BOzY+LSAKIyyJ/fFeA6oJ0ZhdneB1FRGTCxV2Rdw0M8UJDB1ctKNGdDkUkIcRdka/f385oyHFlTbHXUUREomJcRW5m/8vMdptZyMxqIxVqPDY0dJA5OYWFpdleRxERiYrxjsh3Ae8G1kcgS0RsONjBiqo8UpLj7i8bIiKva1xt55zb65yrj1SY8WrpCdLQ3s9K3elQRBJI1IatZnarmW0xsy1tbW0T8h4bGjoAVOQiklBS3uobzOyPwOvtrPmac+6RM30j59w6YB1AbW2tO+OEZ2FDQweZU1KomZ41EYcXEYlJb1nkzrnLoxEkEjY0dHJBVR7Juve4iCSQuLkiGOgO0qj5cRFJQONdfni9mTUDq4D/MbMnIhPr7G1s1Py4iCSmt5xaeTPOuYeBhyOUZVxeONhB1pQU5k3T/LiIJJa4mVrZ0NDBiqp8zY+LSMKJiyI/3n2Spo4BVlbneR1FRCTq4qLINzV2ApofF5HEFBdFvud4D6nJScwpyfQ6iohI1MVFkdcd7+W8ogwm6f4qIpKA4qL56gO9zNNoXEQSlO+L/ET/EIGeoKZVRCRh+b7I6wK9AMzV+nERSVC+L/L6QA+AplZEJGH5vsjrAr3kpk2iMHOy11FERDwRF0U+tyRLD1oWkYTl6yIPhRz7Wnp1oVNEEpqvi/zIiQEGhkaZN01FLiKJy9dFvvf42IqVEq1YEZHE5esirw/0YgazizUiF5HE5esirwv0UJmfztTUZK+jiIh4xudF3sscjcZFJMH5tshPDo3S1NHPXF3oFJEE59si39fSi3O60Cki4tsir3/5HitaQy4iCc63Rd7U0U9KklGel+Z1FBERT/m2yAM9QYqzpuhhyyKS8Hxb5C09QYqydKMsEREfF/kgJVlTvI4hIuI5/xZ5d3hqRUQk0fmyyPsHR+gdHFGRi4jg0yJv6QkCUJKtOXIREV8WeWCsyDUiFxHxaZG3qMhFRE7xaZEPAmjViogIPi3yQHeQzMkppE9O8TqKiIjnfFnk2gwkIvIK3xZ5SbamVUREwLdFPqgLnSIiY8ZV5Gb2H2ZWZ2Y7zOxhM8uJUK43FAo5Wnq0q1NE5GXjHZE/BSxwzi0C9gFfGX+kN9c5MMRIyGnFiojImHEVuXPuSefcyNinG4Cy8Ud6c4FurSEXETldJOfIPwz84Y2+aGa3mtkWM9vS1tZ2zm/yymYgrVoREQF4y4XYZvZHoOR1vvQ159wjY9/zNWAE+OUbHcc5tw5YB1BbW+vOKS2nbQbSqhUREeAMitw5d/mbfd3M/gFYC6x2zp1zQZ+pQE8QMyjM0IhcRATOoMjfjJmtAb4EXOKcG4hMpDfX0h2kIGMyKcm+XDkpIhJx423DO4BM4Ckz225mP45ApjfV0hvUihURkdOMa0TunJsZqSBnKtAdpCw3LdpvKyISs3w3PxHeDKT5cRGRl/mqyIPDo5wYGNbUiojIaXxV5G294aWHxVp6KCJyiq+KXI94ExF5LX8V+dj2fE2tiIi8wldF/vL2fBW5iMgrfFfkk1OSyJqqR7yJiLzMV0V+XmEG1y0pxcy8jiIiEjN8NbS9YUUFN6yo8DqGiEhM8dWIXEREXktFLiLicypyERGfU5GLiPicilxExOdU5CIiPqciFxHxORW5iIjPWRSel/zaNzVrAw6dxb9SALRPUJxYpvNOLIl63pC453625z3DOVf46hc9KfKzZWZbnHO1XueINp13YknU84bEPfdInbemVkREfE5FLiLic34p8nVeB/CIzjuxJOp5Q+Kee0TO2xdz5CIi8sb8MiIXEZE3oCIXEfG5mC9yM1tjZvVmdsDMvux1nmgws3Ize8bM9pjZbjP7tNeZosnMks3sRTN71Oss0WJmOWb2oJnVmdleM1vldaZoMLPPjv2M7zKze80sLh/Ia2Y/NbNWM9t12mt5ZvaUme0f+zP3XI8f00VuZsnAfwLvBGqAG82sxttUUTECfN45VwOsBP53gpz3yz4N7PU6RJT9EHjcOTcXWEwCnL+ZlQK3AbXOuQVAMnCDt6kmzN3Amle99mXgT865WcCfxj4/JzFd5MAK4IBzrsE5NwTcB1zrcaYJ55w77pzbNvZxL+Ff6lJvU0WHmZUBVwM/8TpLtJhZNnAxcBeAc27IOdflaajoSQGmmlkKkAYc8zjPhHDOrQc6X/XytcA9Yx/fA1x3rseP9SIvBY6c9nkzCVJoLzOzSmApsNHjKNHyA+BLQMjjHNFUBbQB/z02pfQTM0v3OtREc84dBb4LHAaOA93OuSe9TRVVxc6542MfB4Dicz1QrBd5QjOzDOA3wGeccz1e55loZrYWaHXObfU6S5SlAMuA/3LOLQX6Gcdfs/1ibE74WsL/I5sOpJvZB7xN5Q0XXgd+zmvBY73IjwLlp31eNvZa3DOzSYRL/JfOuYe8zhMlFwLvMrMmwtNol5nZL7yNFBXNQLNz7uW/dT1IuNjj3eVAo3OuzTk3DDwEvM3jTNHUYmbTAMb+bD3XA8V6kW8GZplZlZmlEr4Q8juPM004MzPC86V7nXPf8zpPtDjnvuKcK3POVRL+b/20cy7uR2jOuQBwxMzmjL20GtjjYaRoOQysNLO0sZ/51STARd7T/A64eezjm4FHzvVAKRGJM0GccyNm9kngCcJXtH/qnNvtcaxouBD4ILDTzLaPvfZV59xj3kWSCfYp4JdjA5YG4BaP80w459xGM3sQ2EZ4pdaLxOlWfTO7F7gUKDCzZuCbwHeA+83sI4Rv6/3ecz6+tuiLiPhbrE+tiIjIW1CRi4j4nIpcRMTnVOQiIj6nIhcR8TkVuYiIz6nIRUR87v8DDCLyf3kZllgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 这个单元格的代码有问题（x包含负数，log未定义）\n",
    "# 已删除，保留词频分析的可视化代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 随机采样示例\n",
    "\n",
    "让我们用一个简单的数字序列来演示随机采样的工作原理：\n",
    "\n",
    "**观察要点**：\n",
    "- 每个批次中的样本是随机选择的子序列\n",
    "- 样本之间没有顺序关系\n",
    "- 适合训练初期，快速探索数据分布\n",
    "\n",
    "**数字示例说明**（假设原始序列为0-34，batch_size=2，num_steps=5）：\n",
    "\n",
    "**第一个批次**：\n",
    "```\n",
    "X: tensor([[14, 15, 16, 17, 18],    # 第一个样本（第0行）\n",
    "           [ 9, 10, 11, 12, 13]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第二个批次**：\n",
    "```\n",
    "X: tensor([[19, 20, 21, 22, 23],    # 第一个样本（第0行）\n",
    "           [29, 30, 31, 32, 33]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第三个批次**：\n",
    "```\n",
    "X: tensor([[ 4,  5,  6,  7,  8],    # 第一个样本（第0行）\n",
    "           [24, 25, 26, 27, 28]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**关键观察**：\n",
    "\n",
    "1. **每个样本（每行）内部是连续的**：\n",
    "   - 第一个样本：14→15→16→17→18（连续）\n",
    "   - 第二个样本：9→10→11→12→13（连续）\n",
    "\n",
    "2. **批次之间没有顺序关系**：\n",
    "   - 第一个批次：14-18和9-13\n",
    "   - 第二个批次：19-23和29-33\n",
    "   - 第三个批次：4-8和24-28\n",
    "   - 注意：第三个批次的4-8实际上在原始序列中位于第一个批次之前，但顺序被打乱了\n",
    "\n",
    "3. **样本顺序被打乱**：\n",
    "   - 原始序列的顺序被破坏\n",
    "   - 样本之间相互独立，没有依赖关系\n",
    "   - 这有利于训练稳定，但不利于学习长距离依赖\n",
    "\n",
    "**工作原理**：\n",
    "- 随机采样首先将所有可能的子序列起始位置打乱（`random.shuffle(initial_indices)`）\n",
    "- 然后按打乱后的顺序生成批次\n",
    "- 每个样本内部保持连续，但样本之间的顺序关系被破坏\n",
    "- 这种设计使样本之间独立，有利于训练稳定，但破坏了序列的连续性\n",
    "\n",
    "**应用场景**：\n",
    "- 训练初期：快速探索数据分布\n",
    "- 数据量很大：需要打乱数据顺序\n",
    "- 模型较简单：不需要长距离依赖\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 随机采样函数定义\n",
    "def seq_data_iter_random(corpus, batch_size, num_steps):\n",
    "    \"\"\"使用随机抽样生成一个小批量子序列\"\"\"\n",
    "    corpus = corpus[random.randint(0, num_steps - 1):]\n",
    "    num_subseqs = (len(corpus) - 1) // num_steps\n",
    "    initial_indices = list(range(0, num_subseqs * num_steps, num_steps))\n",
    "    random.shuffle(initial_indices)\n",
    "\n",
    "    def data(pos):\n",
    "        return corpus[pos: pos + num_steps]\n",
    "\n",
    "    num_batches = num_subseqs // batch_size\n",
    "    for i in range(0, batch_size * num_batches, batch_size):\n",
    "        initial_indices_per_batch = initial_indices[i: i + batch_size]\n",
    "        X = [data(j) for j in initial_indices_per_batch]\n",
    "        Y = [data(j + 1) for j in initial_indices_per_batch]\n",
    "        yield torch.tensor(X), torch.tensor(Y)\n",
    "\n",
    "# 创建一个简单的数字序列用于演示\n",
    "my_seq = list(range(35))\n",
    "\n",
    "# 随机采样示例\n",
    "print(\"随机采样示例（batch_size=2, num_steps=5）：\")\n",
    "print(\"=\" * 60)\n",
    "for X, Y in seq_data_iter_random(my_seq, batch_size=2, num_steps=5):\n",
    "    print('X: ', X, '\\nY:', Y)\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:07.525774Z",
     "iopub.status.busy": "2022-07-31T02:26:07.525101Z",
     "iopub.status.idle": "2022-07-31T02:26:08.377699Z",
     "shell.execute_reply": "2022-07-31T02:26:08.377029Z"
    },
    "origin_pos": 6,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"247.978125pt\" height=\"180.65625pt\" viewBox=\"0 0 247.978125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-07-31T02:26:08.235654</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 247.978125 180.65625 \n",
       "L 247.978125 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 240.778125 143.1 \n",
       "L 240.778125 7.2 \n",
       "L 45.478125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 54.355398 143.1 \n",
       "L 54.355398 7.2 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mf77f89f52e\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf77f89f52e\" x=\"54.355398\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\n",
       "      <g transform=\"translate(45.555398 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 102.85613 143.1 \n",
       "L 102.85613 7.2 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf77f89f52e\" x=\"102.85613\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\n",
       "      <g transform=\"translate(94.05613 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 151.356861 143.1 \n",
       "L 151.356861 7.2 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf77f89f52e\" x=\"151.356861\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\n",
       "      <g transform=\"translate(142.556861 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 199.857593 143.1 \n",
       "L 199.857593 7.2 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf77f89f52e\" x=\"199.857593\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- $\\mathdefault{10^{3}}$ -->\n",
       "      <g transform=\"translate(191.057593 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <defs>\n",
       "       <path id=\"m3f45ff7a80\" d=\"M 0 0 \n",
       "L 0 2 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"46.842539\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"49.655191\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"52.136126\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"68.955573\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"77.496128\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"83.555748\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_11\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"88.255954\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_12\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"92.096303\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_13\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"95.343271\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_14\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"98.155923\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_15\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"100.636858\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_16\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"117.456305\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_17\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"125.99686\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_18\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"132.05648\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_19\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"136.756686\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_20\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"140.597035\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_21\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"143.844003\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_22\">\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"146.656655\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_23\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"149.13759\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_24\">\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"165.957036\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_25\">\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"174.497591\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_26\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"180.557211\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_27\">\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"185.257418\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_28\">\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"189.097766\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_29\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"192.344735\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_30\">\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"195.157387\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_31\">\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"197.638321\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_32\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"214.457768\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_33\">\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"222.998323\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_34\">\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"229.057943\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_35\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"233.75815\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_36\">\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3f45ff7a80\" x=\"237.598498\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- token: x -->\n",
       "     <g transform=\"translate(122.916406 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-3a\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 0 \n",
       "L 750 0 \n",
       "L 750 794 \n",
       "z\n",
       "M 750 3309 \n",
       "L 1409 3309 \n",
       "L 1409 2516 \n",
       "L 750 2516 \n",
       "L 750 3309 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"100.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"154.675781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"216.199219\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-3a\" x=\"279.578125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"313.269531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"345.056641\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_41\">\n",
       "      <path d=\"M 45.478125 136.922727 \n",
       "L 240.778125 136.922727 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_42\">\n",
       "      <defs>\n",
       "       <path id=\"m55b4dd4953\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55b4dd4953\" x=\"45.478125\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_43\">\n",
       "      <path d=\"M 45.478125 100.09077 \n",
       "L 240.778125 100.09077 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55b4dd4953\" x=\"45.478125\" y=\"100.09077\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\n",
       "      <g transform=\"translate(20.878125 103.889989)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_45\">\n",
       "      <path d=\"M 45.478125 63.258813 \n",
       "L 240.778125 63.258813 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_46\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55b4dd4953\" x=\"45.478125\" y=\"63.258813\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\n",
       "      <g transform=\"translate(20.878125 67.058032)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_47\">\n",
       "      <path d=\"M 45.478125 26.426856 \n",
       "L 240.778125 26.426856 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_48\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m55b4dd4953\" x=\"45.478125\" y=\"26.426856\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- $\\mathdefault{10^{3}}$ -->\n",
       "      <g transform=\"translate(20.878125 30.226075)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_49\">\n",
       "      <defs>\n",
       "       <path id=\"mebfe2e87fb\" d=\"M 0 0 \n",
       "L -2 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"142.62807\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_50\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"140.492113\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_51\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"138.608065\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_52\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"125.835203\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_53\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"119.349418\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_54\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"114.747679\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_55\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"111.178294\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_56\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"108.261894\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_13\">\n",
       "     <g id=\"line2d_57\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"105.796112\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_14\">\n",
       "     <g id=\"line2d_58\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"103.660156\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_15\">\n",
       "     <g id=\"line2d_59\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"101.776108\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_16\">\n",
       "     <g id=\"line2d_60\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"89.003246\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_17\">\n",
       "     <g id=\"line2d_61\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"82.517461\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_18\">\n",
       "     <g id=\"line2d_62\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"77.915722\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_19\">\n",
       "     <g id=\"line2d_63\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"74.346337\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_20\">\n",
       "     <g id=\"line2d_64\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"71.429937\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_21\">\n",
       "     <g id=\"line2d_65\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"68.964155\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_22\">\n",
       "     <g id=\"line2d_66\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"66.828198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_23\">\n",
       "     <g id=\"line2d_67\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"64.944151\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_24\">\n",
       "     <g id=\"line2d_68\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"52.171289\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_25\">\n",
       "     <g id=\"line2d_69\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"45.685503\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_26\">\n",
       "     <g id=\"line2d_70\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"41.083765\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_27\">\n",
       "     <g id=\"line2d_71\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"37.51438\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_28\">\n",
       "     <g id=\"line2d_72\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"34.597979\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_29\">\n",
       "     <g id=\"line2d_73\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"32.132198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_30\">\n",
       "     <g id=\"line2d_74\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"29.996241\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_31\">\n",
       "     <g id=\"line2d_75\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"28.112194\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_32\">\n",
       "     <g id=\"line2d_76\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"15.339332\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_33\">\n",
       "     <g id=\"line2d_77\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mebfe2e87fb\" x=\"45.478125\" y=\"8.853546\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- frequency: n(x) -->\n",
       "     <g transform=\"translate(14.798438 113.167187)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-71\" d=\"M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "M 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 -1331 \n",
       "L 2906 -1331 \n",
       "L 2906 525 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"74.068359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-71\" x=\"135.591797\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"199.068359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"262.447266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"323.970703\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"387.349609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"442.330078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-3a\" x=\"494.259766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"527.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"559.738281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"623.117188\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"662.130859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"721.310547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_78\">\n",
       "    <path d=\"M -1 22.630819 \n",
       "L 54.355398 22.641392 \n",
       "L 68.955573 22.921582 \n",
       "L 77.496128 24.121839 \n",
       "L 83.555748 29.67948 \n",
       "L 88.255954 32.246865 \n",
       "L 92.096303 35.931745 \n",
       "L 95.343271 36.253723 \n",
       "L 98.155923 39.450499 \n",
       "L 102.85613 39.668628 \n",
       "L 104.863705 43.037948 \n",
       "L 106.696478 46.732081 \n",
       "L 108.382465 47.370841 \n",
       "L 111.396684 50.574172 \n",
       "L 112.756098 50.940227 \n",
       "L 114.033071 51.854528 \n",
       "L 115.237033 52.171289 \n",
       "L 116.375883 55.941884 \n",
       "L 117.456305 56.561158 \n",
       "L 118.484001 58.223127 \n",
       "L 119.46388 58.223127 \n",
       "L 120.400194 58.577295 \n",
       "L 121.296653 59.185578 \n",
       "L 122.156511 59.435519 \n",
       "L 122.982641 59.947432 \n",
       "L 123.777588 60.078011 \n",
       "L 125.282771 60.078011 \n",
       "L 126.687531 60.611258 \n",
       "L 127.356273 61.162896 \n",
       "L 128.004435 61.162896 \n",
       "L 128.633246 61.30383 \n",
       "L 129.243828 61.30383 \n",
       "L 129.837208 61.446017 \n",
       "L 130.414329 62.326748 \n",
       "L 130.976058 62.47837 \n",
       "L 131.523195 62.785993 \n",
       "L 132.05648 64.248567 \n",
       "L 133.084176 64.592578 \n",
       "L 133.579813 64.592578 \n",
       "L 134.064055 64.767399 \n",
       "L 134.537414 65.122878 \n",
       "L 136.331144 65.858452 \n",
       "L 137.173801 65.858452 \n",
       "L 137.582816 66.047755 \n",
       "L 137.984039 66.047755 \n",
       "L 138.377763 66.433217 \n",
       "L 139.143796 67.439582 \n",
       "L 139.516613 67.860551 \n",
       "L 139.882946 68.075265 \n",
       "L 140.597035 68.075265 \n",
       "L 141.956448 68.964155 \n",
       "L 142.60461 68.964155 \n",
       "L 142.921362 69.194316 \n",
       "L 143.233421 69.905363 \n",
       "L 143.540924 70.905433 \n",
       "L 144.437383 71.698782 \n",
       "L 144.72792 71.698782 \n",
       "L 145.014504 72.25042 \n",
       "L 145.576233 72.25042 \n",
       "L 145.851578 72.821763 \n",
       "L 146.123371 73.115275 \n",
       "L 146.3917 73.115275 \n",
       "L 147.684351 74.669498 \n",
       "L 148.90224 74.669498 \n",
       "L 149.13759 74.999322 \n",
       "L 149.370339 74.999322 \n",
       "L 149.600544 75.336091 \n",
       "L 149.828261 75.336091 \n",
       "L 150.053543 75.680102 \n",
       "L 150.715281 75.680102 \n",
       "L 150.93132 76.031675 \n",
       "L 151.145165 76.031675 \n",
       "L 151.566451 76.758887 \n",
       "L 151.979476 76.758887 \n",
       "L 152.182991 77.135279 \n",
       "L 152.384558 77.135279 \n",
       "L 152.781996 77.915722 \n",
       "L 153.364437 77.915722 \n",
       "L 153.743971 78.736206 \n",
       "L 153.931205 78.736206 \n",
       "L 154.116788 79.162789 \n",
       "L 154.843191 79.162789 \n",
       "L 155.020944 79.60106 \n",
       "L 155.887881 79.60106 \n",
       "L 156.057068 80.051679 \n",
       "L 156.720543 80.051679 \n",
       "L 156.883197 80.515361 \n",
       "L 157.678144 80.515361 \n",
       "L 157.833596 80.992887 \n",
       "L 158.141099 80.992887 \n",
       "L 158.293184 81.485108 \n",
       "L 158.890773 81.485108 \n",
       "L 159.037558 81.992957 \n",
       "L 159.328095 81.992957 \n",
       "L 159.471874 82.517461 \n",
       "L 159.756522 82.517461 \n",
       "L 159.897416 83.059747 \n",
       "L 160.037374 83.059747 \n",
       "L 160.176408 83.621065 \n",
       "L 160.858138 83.621065 \n",
       "L 160.991875 84.202798 \n",
       "L 161.518493 84.202798 \n",
       "L 161.648115 84.80649 \n",
       "L 162.409533 84.80649 \n",
       "L 162.533803 85.433861 \n",
       "L 162.902271 85.433861 \n",
       "L 163.023676 86.086846 \n",
       "L 163.502415 86.086846 \n",
       "L 163.620418 86.767626 \n",
       "L 164.200719 86.767626 \n",
       "L 164.314886 87.478673 \n",
       "L 165.423752 87.478673 \n",
       "L 165.638689 88.222803 \n",
       "L 166.374151 88.222803 \n",
       "L 166.579651 89.003246 \n",
       "L 167.675405 89.003246 \n",
       "L 167.86865 89.823729 \n",
       "L 168.716963 89.823729 \n",
       "L 168.900926 90.688584 \n",
       "L 169.173892 90.688584 \n",
       "L 169.353924 91.602885 \n",
       "L 169.972187 91.602885 \n",
       "L 170.145551 92.572632 \n",
       "L 171.483372 92.572632 \n",
       "L 171.64478 93.604984 \n",
       "L 172.433771 93.604984 \n",
       "L 172.588084 94.708589 \n",
       "L 173.490948 94.708589 \n",
       "L 173.637733 95.894013 \n",
       "L 175.256101 95.894013 \n",
       "L 175.391124 97.17437 \n",
       "L 176.37712 97.17437 \n",
       "L 176.505167 98.566197 \n",
       "L 178.33794 98.566197 \n",
       "L 178.454636 100.09077 \n",
       "L 180.451629 100.09077 \n",
       "L 180.609805 101.776108 \n",
       "L 182.42068 101.776108 \n",
       "L 182.564787 103.660156 \n",
       "L 185.257418 103.660156 \n",
       "L 185.383422 105.796112 \n",
       "L 188.528445 105.796112 \n",
       "L 188.672225 108.261894 \n",
       "L 191.67211 108.261894 \n",
       "L 191.796014 111.178294 \n",
       "L 195.754423 111.178294 \n",
       "L 195.882004 114.747679 \n",
       "L 200.724191 114.747679 \n",
       "L 200.84513 119.349418 \n",
       "L 207.214001 119.349418 \n",
       "L 207.332503 125.835203 \n",
       "L 216.282641 125.835203 \n",
       "L 216.398216 136.922727 \n",
       "L 231.900852 136.922727 \n",
       "L 231.900852 136.922727 \n",
       "\" clip-path=\"url(#pf2890a3df7)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 45.478125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 240.778125 143.1 \n",
       "L 240.778125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 240.778125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 45.478125 7.2 \n",
       "L 240.778125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pf2890a3df7\">\n",
       "   <rect x=\"45.478125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "freqs = [freq for token, freq in vocab.token_freqs]\n",
    "d2l.plot(freqs, xlabel='token: x', ylabel='frequency: n(x)',\n",
    "         xscale='log', yscale='log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 顺序采样示例\n",
    "\n",
    "**观察要点**：\n",
    "- 每个批次中的样本在原始序列上是连续的\n",
    "- 相邻批次之间也保持连续性\n",
    "- 第一行的样本：4→9→14（连续）\n",
    "- 第二行的样本：19→24→29（连续）\n",
    "\n",
    "**数字示例说明**（假设原始序列为0-34，batch_size=2，num_steps=5）：\n",
    "\n",
    "**第一个批次**：\n",
    "```\n",
    "X: tensor([[ 4,  5,  6,  7,  8],    # 第一个样本（第0行）\n",
    "           [19, 20, 21, 22, 23]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**第二个批次**：\n",
    "```\n",
    "X: tensor([[ 9, 10, 11, 12, 13],    # 第一个样本（第0行）\n",
    "           [24, 25, 26, 27, 28]])   # 第二个样本（第1行）\n",
    "```\n",
    "\n",
    "**关键观察**：\n",
    "\n",
    "1. **每个样本（每行）内部是连续的**：\n",
    "   - 第一个样本：4→5→6→7→8（连续）\n",
    "   - 第二个样本：19→20→21→22→23（连续）\n",
    "\n",
    "2. **同一批次内不同样本（不同行）之间不连续**：\n",
    "   - 第一个样本从位置4开始\n",
    "   - 第二个样本从位置19开始\n",
    "   - 它们之间间隔了`batch_size=2`个位置（实际间隔更大，因为它们是并行序列）\n",
    "\n",
    "3. **相邻批次之间每个样本（每行）是连续的**：\n",
    "   - 第一个样本：4→9→14（跨批次连续）\n",
    "   - 第二个样本：19→24→29（跨批次连续）\n",
    "\n",
    "**工作原理**：\n",
    "- 顺序采样通过`reshape(batch_size, -1)`将序列重塑为`batch_size`个并行序列\n",
    "- 每个并行序列在原始序列中每隔`batch_size`个位置采样一个词元\n",
    "- 每个并行序列内部是连续的，不同并行序列之间间隔`batch_size`个位置\n",
    "- 这种设计允许并行处理多个序列（提高效率），同时保持每个序列的连续性（有利于RNN学习）\n",
    "\n",
    "**应用场景**：\n",
    "- RNN/LSTM训练：利用隐藏状态的连续性\n",
    "- 长文本建模：保持上下文信息\n",
    "- 序列到序列任务：编码器-解码器架构\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 顺序采样函数定义\n",
    "def seq_data_iter_sequential(corpus, batch_size, num_steps):\n",
    "    \"\"\"使用顺序分区生成一个小批量子序列\"\"\"\n",
    "    offset = random.randint(0, num_steps)\n",
    "    num_tokens = ((len(corpus) - offset - 1) // batch_size) * batch_size\n",
    "    Xs = torch.tensor(corpus[offset: offset + num_tokens])\n",
    "    Ys = torch.tensor(corpus[offset + 1: offset + 1 + num_tokens])\n",
    "    Xs, Ys = Xs.reshape(batch_size, -1), Ys.reshape(batch_size, -1)\n",
    "    num_batches = Xs.shape[1] // num_steps\n",
    "    for i in range(0, num_steps * num_batches, num_steps):\n",
    "        X = Xs[:, i: i + num_steps]\n",
    "        Y = Ys[:, i: i + num_steps]\n",
    "        yield X, Y\n",
    "\n",
    "# 创建一个简单的数字序列用于演示\n",
    "my_seq = list(range(35))\n",
    "\n",
    "# 顺序采样示例\n",
    "print(\"顺序采样示例（batch_size=2, num_steps=5）：\")\n",
    "print(\"=\" * 60)\n",
    "for X, Y in seq_data_iter_sequential(my_seq, batch_size=2, num_steps=5):\n",
    "    print('X: ', X, '\\nY:', Y)\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:08.381292Z",
     "iopub.status.busy": "2022-07-31T02:26:08.380775Z",
     "iopub.status.idle": "2022-07-31T02:26:08.413107Z",
     "shell.execute_reply": "2022-07-31T02:26:08.412466Z"
    },
    "origin_pos": 8,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('of', 'the'), 309),\n",
       " (('in', 'the'), 169),\n",
       " (('i', 'had'), 130),\n",
       " (('i', 'was'), 112),\n",
       " (('and', 'the'), 109),\n",
       " (('the', 'time'), 102),\n",
       " (('it', 'was'), 99),\n",
       " (('to', 'the'), 85),\n",
       " (('as', 'i'), 78),\n",
       " (('of', 'a'), 73)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_tokens = [pair for pair in zip(corpus[:-1], corpus[1:])]\n",
    "bigram_vocab = d2l.Vocab(bigram_tokens)\n",
    "bigram_vocab.token_freqs[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:08.415996Z",
     "iopub.status.busy": "2022-07-31T02:26:08.415774Z",
     "iopub.status.idle": "2022-07-31T02:26:08.455773Z",
     "shell.execute_reply": "2022-07-31T02:26:08.455082Z"
    },
    "origin_pos": 10,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('the', 'time', 'traveller'), 59),\n",
       " (('the', 'time', 'machine'), 30),\n",
       " (('the', 'medical', 'man'), 24),\n",
       " (('it', 'seemed', 'to'), 16),\n",
       " (('it', 'was', 'a'), 15),\n",
       " (('here', 'and', 'there'), 15),\n",
       " (('seemed', 'to', 'me'), 14),\n",
       " (('i', 'did', 'not'), 14),\n",
       " (('i', 'saw', 'the'), 13),\n",
       " (('i', 'began', 'to'), 13)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trigram_tokens = [triple for triple in zip(\n",
    "    corpus[:-2], corpus[1:-1], corpus[2:])]\n",
    "trigram_vocab = d2l.Vocab(trigram_tokens)\n",
    "trigram_vocab.token_freqs[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:08.458631Z",
     "iopub.status.busy": "2022-07-31T02:26:08.458400Z",
     "iopub.status.idle": "2022-07-31T02:26:09.197937Z",
     "shell.execute_reply": "2022-07-31T02:26:09.197265Z"
    },
    "origin_pos": 12,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"247.978125pt\" height=\"180.65625pt\" viewBox=\"0 0 247.978125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-07-31T02:26:09.010422</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 247.978125 180.65625 \n",
       "L 247.978125 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 240.778125 143.1 \n",
       "L 240.778125 7.2 \n",
       "L 45.478125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 54.355398 143.1 \n",
       "L 54.355398 7.2 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m05ab0ed134\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05ab0ed134\" x=\"54.355398\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\n",
       "      <g transform=\"translate(45.555398 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 94.026857 143.1 \n",
       "L 94.026857 7.2 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05ab0ed134\" x=\"94.026857\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\n",
       "      <g transform=\"translate(85.226857 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 133.698316 143.1 \n",
       "L 133.698316 7.2 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05ab0ed134\" x=\"133.698316\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\n",
       "      <g transform=\"translate(124.898316 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 173.369775 143.1 \n",
       "L 173.369775 7.2 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05ab0ed134\" x=\"173.369775\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- $\\mathdefault{10^{3}}$ -->\n",
       "      <g transform=\"translate(164.569775 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 213.041234 143.1 \n",
       "L 213.041234 7.2 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05ab0ed134\" x=\"213.041234\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- $\\mathdefault{10^{4}}$ -->\n",
       "      <g transform=\"translate(204.241234 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <defs>\n",
       "       <path id=\"m0be796e049\" d=\"M 0 0 \n",
       "L 0 2 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"45.554334\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"48.210211\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"50.510836\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"52.540131\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"66.297697\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_11\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"73.283494\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_12\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"78.239996\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_13\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"82.084558\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_14\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"85.225793\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_15\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"87.88167\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_16\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"90.182295\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_17\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"92.21159\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_18\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"105.969156\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_19\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"112.954953\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_20\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"117.911455\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_21\">\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"121.756017\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_22\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"124.897252\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_23\">\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"127.553129\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_24\">\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"129.853754\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_25\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"131.883049\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_26\">\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"145.640615\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_27\">\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"152.626412\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_28\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"157.582914\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_29\">\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"161.427476\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_30\">\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"164.568711\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_31\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"167.224588\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_32\">\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"169.525213\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_33\">\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"171.554508\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_34\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"185.312074\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_35\">\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"192.297871\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_36\">\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"197.254373\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_37\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"201.098935\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_38\">\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"204.24017\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_39\">\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"206.896047\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_40\">\n",
       "     <g id=\"line2d_45\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"209.196672\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_41\">\n",
       "     <g id=\"line2d_46\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"211.225968\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_42\">\n",
       "     <g id=\"line2d_47\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"224.983533\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_43\">\n",
       "     <g id=\"line2d_48\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"231.96933\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_44\">\n",
       "     <g id=\"line2d_49\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"236.925832\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_45\">\n",
       "     <g id=\"line2d_50\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0be796e049\" x=\"240.770394\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- token: x -->\n",
       "     <g transform=\"translate(122.916406 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-3a\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 0 \n",
       "L 750 0 \n",
       "L 750 794 \n",
       "z\n",
       "M 750 3309 \n",
       "L 1409 3309 \n",
       "L 1409 2516 \n",
       "L 750 2516 \n",
       "L 750 3309 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6b\" x=\"100.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"154.675781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"216.199219\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-3a\" x=\"279.578125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"313.269531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"345.056641\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_51\">\n",
       "      <path d=\"M 45.478125 136.922727 \n",
       "L 240.778125 136.922727 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_52\">\n",
       "      <defs>\n",
       "       <path id=\"m57bc4e1f9e\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m57bc4e1f9e\" x=\"45.478125\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- $\\mathdefault{10^{0}}$ -->\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_53\">\n",
       "      <path d=\"M 45.478125 100.09077 \n",
       "L 240.778125 100.09077 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_54\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m57bc4e1f9e\" x=\"45.478125\" y=\"100.09077\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- $\\mathdefault{10^{1}}$ -->\n",
       "      <g transform=\"translate(20.878125 103.889989)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_55\">\n",
       "      <path d=\"M 45.478125 63.258813 \n",
       "L 240.778125 63.258813 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_56\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m57bc4e1f9e\" x=\"45.478125\" y=\"63.258813\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- $\\mathdefault{10^{2}}$ -->\n",
       "      <g transform=\"translate(20.878125 67.058032)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_57\">\n",
       "      <path d=\"M 45.478125 26.426856 \n",
       "L 240.778125 26.426856 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_58\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m57bc4e1f9e\" x=\"45.478125\" y=\"26.426856\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- $\\mathdefault{10^{3}}$ -->\n",
       "      <g transform=\"translate(20.878125 30.226075)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_59\">\n",
       "      <defs>\n",
       "       <path id=\"mff8e8ed9de\" d=\"M 0 0 \n",
       "L -2 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"142.62807\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_60\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"140.492113\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_61\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"138.608065\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_62\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"125.835203\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_63\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"119.349418\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_64\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"114.747679\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_65\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"111.178294\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_66\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"108.261894\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_13\">\n",
       "     <g id=\"line2d_67\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"105.796112\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_14\">\n",
       "     <g id=\"line2d_68\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"103.660156\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_15\">\n",
       "     <g id=\"line2d_69\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"101.776108\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_16\">\n",
       "     <g id=\"line2d_70\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"89.003246\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_17\">\n",
       "     <g id=\"line2d_71\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"82.517461\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_18\">\n",
       "     <g id=\"line2d_72\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"77.915722\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_19\">\n",
       "     <g id=\"line2d_73\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"74.346337\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_20\">\n",
       "     <g id=\"line2d_74\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"71.429937\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_21\">\n",
       "     <g id=\"line2d_75\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"68.964155\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_22\">\n",
       "     <g id=\"line2d_76\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"66.828198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_23\">\n",
       "     <g id=\"line2d_77\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"64.944151\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_24\">\n",
       "     <g id=\"line2d_78\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"52.171289\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_25\">\n",
       "     <g id=\"line2d_79\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"45.685503\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_26\">\n",
       "     <g id=\"line2d_80\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"41.083765\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_27\">\n",
       "     <g id=\"line2d_81\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"37.51438\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_28\">\n",
       "     <g id=\"line2d_82\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"34.597979\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_29\">\n",
       "     <g id=\"line2d_83\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"32.132198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_30\">\n",
       "     <g id=\"line2d_84\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"29.996241\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_31\">\n",
       "     <g id=\"line2d_85\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"28.112194\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_32\">\n",
       "     <g id=\"line2d_86\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"15.339332\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_33\">\n",
       "     <g id=\"line2d_87\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mff8e8ed9de\" x=\"45.478125\" y=\"8.853546\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- frequency: n(x) -->\n",
       "     <g transform=\"translate(14.798438 113.167187)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-71\" d=\"M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "M 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 -1331 \n",
       "L 2906 -1331 \n",
       "L 2906 525 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"35.205078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"74.068359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-71\" x=\"135.591797\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"199.068359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"262.447266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"323.970703\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"387.349609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"442.330078\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-3a\" x=\"494.259766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"527.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"559.738281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"623.117188\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"662.130859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"721.310547\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_88\">\n",
       "    <path d=\"M -1 22.628465 \n",
       "L 54.355398 22.641392 \n",
       "L 66.297697 22.921582 \n",
       "L 73.283494 24.121839 \n",
       "L 78.239996 29.67948 \n",
       "L 82.084558 32.246865 \n",
       "L 85.225793 35.931745 \n",
       "L 87.88167 36.253723 \n",
       "L 90.182295 39.450499 \n",
       "L 94.026857 39.668628 \n",
       "L 95.668965 43.037948 \n",
       "L 97.168092 46.732081 \n",
       "L 98.547156 47.370841 \n",
       "L 101.012654 50.574172 \n",
       "L 102.124594 50.940227 \n",
       "L 103.169102 51.854528 \n",
       "L 104.15389 52.171289 \n",
       "L 105.085419 55.941884 \n",
       "L 105.969156 56.561158 \n",
       "L 106.809766 58.223127 \n",
       "L 107.611264 58.223127 \n",
       "L 108.377128 58.577295 \n",
       "L 109.110391 59.185578 \n",
       "L 109.813718 59.435519 \n",
       "L 110.489455 59.947432 \n",
       "L 111.139687 60.078011 \n",
       "L 112.37086 60.078011 \n",
       "L 113.519892 60.611258 \n",
       "L 114.066893 61.162896 \n",
       "L 114.597061 61.162896 \n",
       "L 115.111401 61.30383 \n",
       "L 115.61083 61.30383 \n",
       "L 116.096189 61.446017 \n",
       "L 116.568248 62.326748 \n",
       "L 117.027718 62.47837 \n",
       "L 117.475252 62.785993 \n",
       "L 117.911455 64.248567 \n",
       "L 118.752065 64.592578 \n",
       "L 119.157475 64.592578 \n",
       "L 119.553563 64.767399 \n",
       "L 119.94075 65.122878 \n",
       "L 121.407942 65.858452 \n",
       "L 122.097198 65.858452 \n",
       "L 122.431754 66.047755 \n",
       "L 122.759937 66.047755 \n",
       "L 123.081986 66.433217 \n",
       "L 123.708567 67.439582 \n",
       "L 124.013515 67.860551 \n",
       "L 124.313159 68.075265 \n",
       "L 124.897252 68.075265 \n",
       "L 126.009193 68.964155 \n",
       "L 126.53936 68.964155 \n",
       "L 126.798449 69.194316 \n",
       "L 127.553129 71.165535 \n",
       "L 128.038488 71.698782 \n",
       "L 128.276134 71.698782 \n",
       "L 128.510547 72.25042 \n",
       "L 128.970017 72.25042 \n",
       "L 129.417551 73.115275 \n",
       "L 129.637033 73.115275 \n",
       "L 130.898262 74.669498 \n",
       "L 131.690544 74.669498 \n",
       "L 131.883049 74.999322 \n",
       "L 132.073428 74.999322 \n",
       "L 132.261726 75.336091 \n",
       "L 132.447988 75.336091 \n",
       "L 132.632259 75.680102 \n",
       "L 133.173531 75.680102 \n",
       "L 133.350241 76.031675 \n",
       "L 133.525158 76.031675 \n",
       "L 133.869751 76.758887 \n",
       "L 134.207587 76.758887 \n",
       "L 134.374053 77.135279 \n",
       "L 134.538926 77.135279 \n",
       "L 134.864013 77.915722 \n",
       "L 135.340424 77.915722 \n",
       "L 135.650867 78.736206 \n",
       "L 135.804015 78.736206 \n",
       "L 135.955814 79.162789 \n",
       "L 136.54998 79.162789 \n",
       "L 136.695374 79.60106 \n",
       "L 137.40449 79.60106 \n",
       "L 137.542877 80.051679 \n",
       "L 138.085571 80.051679 \n",
       "L 138.218615 80.515361 \n",
       "L 138.868847 80.515361 \n",
       "L 138.995999 80.992887 \n",
       "L 139.247523 80.992887 \n",
       "L 139.371922 81.485108 \n",
       "L 139.860723 81.485108 \n",
       "L 139.980787 81.992957 \n",
       "L 140.218433 81.992957 \n",
       "L 140.336039 82.517461 \n",
       "L 140.568868 82.517461 \n",
       "L 140.684113 83.059747 \n",
       "L 140.798592 83.059747 \n",
       "L 141.025294 83.621065 \n",
       "L 141.469941 83.621065 \n",
       "L 141.688034 84.202798 \n",
       "L 142.010082 84.202798 \n",
       "L 142.221485 84.80649 \n",
       "L 142.738914 84.80649 \n",
       "L 142.941611 85.433861 \n",
       "L 143.141952 85.433861 \n",
       "L 143.33999 86.086846 \n",
       "L 143.632843 86.086846 \n",
       "L 143.825349 86.767626 \n",
       "L 144.204025 86.767626 \n",
       "L 144.390288 87.478673 \n",
       "L 145.204412 87.478673 \n",
       "L 145.380221 88.222803 \n",
       "L 145.981796 88.222803 \n",
       "L 146.149886 89.003246 \n",
       "L 147.046164 89.003246 \n",
       "L 147.204231 89.823729 \n",
       "L 147.898113 89.823729 \n",
       "L 148.048587 90.688584 \n",
       "L 148.271861 90.688584 \n",
       "L 148.419119 91.602885 \n",
       "L 148.924831 91.602885 \n",
       "L 149.066635 92.572632 \n",
       "L 150.160914 92.572632 \n",
       "L 150.292938 93.604984 \n",
       "L 150.938298 93.604984 \n",
       "L 151.064519 94.708589 \n",
       "L 151.803022 94.708589 \n",
       "L 151.923086 95.894013 \n",
       "L 153.246839 95.894013 \n",
       "L 153.41224 97.17437 \n",
       "L 154.163784 97.17437 \n",
       "L 154.320651 98.566197 \n",
       "L 155.767648 98.566197 \n",
       "L 155.910629 100.09077 \n",
       "L 157.496553 100.09077 \n",
       "L 157.625933 101.776108 \n",
       "L 159.107149 101.776108 \n",
       "L 159.225022 103.660156 \n",
       "L 161.427476 103.660156 \n",
       "L 161.56476 105.796112 \n",
       "L 164.103032 105.796112 \n",
       "L 164.220637 108.261894 \n",
       "L 166.674411 108.261894 \n",
       "L 166.801003 111.178294 \n",
       "L 170.013563 111.178294 \n",
       "L 170.138714 114.747679 \n",
       "L 174.078613 114.747679 \n",
       "L 174.193969 119.349418 \n",
       "L 179.386992 119.349418 \n",
       "L 179.508069 125.835203 \n",
       "L 186.804739 125.835203 \n",
       "L 186.922828 136.922727 \n",
       "L 199.579747 136.922727 \n",
       "L 199.579747 136.922727 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_89\">\n",
       "    <path d=\"M -1 54.851831 \n",
       "L 54.355398 54.8653 \n",
       "L 73.283494 61.446017 \n",
       "L 78.239996 61.880322 \n",
       "L 82.084558 62.942052 \n",
       "L 85.225793 63.419577 \n",
       "L 87.88167 65.858452 \n",
       "L 90.182295 67.23318 \n",
       "L 92.21159 68.2929 \n",
       "L 94.026857 69.427838 \n",
       "L 95.668965 69.664818 \n",
       "L 97.168092 70.905433 \n",
       "L 98.547156 71.165535 \n",
       "L 99.823969 71.165535 \n",
       "L 101.012654 71.429937 \n",
       "L 102.124594 74.029576 \n",
       "L 103.169102 74.029576 \n",
       "L 104.15389 74.669498 \n",
       "L 105.085419 74.669498 \n",
       "L 105.969156 75.336091 \n",
       "L 106.809766 75.336091 \n",
       "L 108.377128 76.031675 \n",
       "L 109.813718 76.031675 \n",
       "L 110.489455 76.391149 \n",
       "L 111.139687 76.391149 \n",
       "L 111.766268 77.520741 \n",
       "L 112.37086 77.915722 \n",
       "L 112.954953 77.915722 \n",
       "L 113.519892 78.320704 \n",
       "L 115.111401 78.320704 \n",
       "L 115.61083 79.162789 \n",
       "L 116.096189 79.60106 \n",
       "L 117.027718 79.60106 \n",
       "L 117.475252 80.515361 \n",
       "L 118.336886 80.515361 \n",
       "L 118.752065 81.485108 \n",
       "L 119.157475 81.485108 \n",
       "L 119.94075 82.517461 \n",
       "L 121.052691 82.517461 \n",
       "L 121.756017 83.621065 \n",
       "L 122.431754 83.621065 \n",
       "L 122.759937 84.202798 \n",
       "L 123.708567 84.202798 \n",
       "L 124.013515 84.80649 \n",
       "L 124.607681 84.80649 \n",
       "L 124.897252 85.433861 \n",
       "L 125.462191 85.433861 \n",
       "L 125.737863 86.086846 \n",
       "L 126.53936 86.086846 \n",
       "L 126.798449 86.767626 \n",
       "L 127.553129 86.767626 \n",
       "L 127.797518 87.478673 \n",
       "L 129.637033 87.478673 \n",
       "L 129.853754 88.222803 \n",
       "L 130.898262 88.222803 \n",
       "L 131.099774 89.003246 \n",
       "L 131.495862 89.003246 \n",
       "L 131.690544 89.823729 \n",
       "L 133.173531 89.823729 \n",
       "L 133.350241 90.688584 \n",
       "L 134.538926 90.688584 \n",
       "L 134.702236 91.602885 \n",
       "L 135.18308 91.602885 \n",
       "L 135.340424 92.572632 \n",
       "L 137.542877 92.572632 \n",
       "L 137.680162 93.604984 \n",
       "L 139.980787 93.604984 \n",
       "L 140.10002 94.708589 \n",
       "L 141.796053 94.708589 \n",
       "L 142.010082 95.894013 \n",
       "L 143.535777 95.894013 \n",
       "L 143.729364 97.17437 \n",
       "L 145.292541 97.17437 \n",
       "L 145.467457 98.566197 \n",
       "L 147.438644 98.566197 \n",
       "L 147.593166 100.09077 \n",
       "L 150.489097 100.09077 \n",
       "L 150.61864 101.776108 \n",
       "L 152.854615 101.776108 \n",
       "L 152.967594 103.660156 \n",
       "L 156.099442 103.660156 \n",
       "L 156.239707 105.796112 \n",
       "L 159.458377 105.796112 \n",
       "L 159.57388 108.261894 \n",
       "L 163.225504 108.261894 \n",
       "L 163.349233 111.178294 \n",
       "L 167.82918 111.178294 \n",
       "L 167.947593 114.747679 \n",
       "L 174.128146 114.747679 \n",
       "L 174.243171 119.349418 \n",
       "L 182.205262 119.349418 \n",
       "L 182.318375 125.835203 \n",
       "L 195.477351 125.835203 \n",
       "L 195.591578 136.922727 \n",
       "L 225.183092 136.922727 \n",
       "L 225.183092 136.922727 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_90\">\n",
       "    <path d=\"M -1 82.502365 \n",
       "L 54.355398 82.517461 \n",
       "L 66.297697 86.086846 \n",
       "L 73.283494 92.572632 \n",
       "L 78.239996 93.604984 \n",
       "L 82.084558 93.604984 \n",
       "L 85.225793 94.708589 \n",
       "L 87.88167 94.708589 \n",
       "L 90.182295 95.894013 \n",
       "L 92.21159 95.894013 \n",
       "L 94.026857 97.17437 \n",
       "L 98.547156 97.17437 \n",
       "L 99.823969 98.566197 \n",
       "L 105.969156 98.566197 \n",
       "L 106.809766 100.09077 \n",
       "L 109.813718 100.09077 \n",
       "L 110.489455 101.776108 \n",
       "L 114.066893 101.776108 \n",
       "L 114.597061 103.660156 \n",
       "L 121.052691 103.660156 \n",
       "L 121.407942 105.796112 \n",
       "L 126.276316 105.796112 \n",
       "L 126.53936 108.261894 \n",
       "L 132.632259 108.261894 \n",
       "L 132.814579 111.178294 \n",
       "L 140.568868 111.178294 \n",
       "L 140.684113 114.747679 \n",
       "L 148.271861 114.747679 \n",
       "L 148.419119 119.349418 \n",
       "L 160.652212 119.349418 \n",
       "L 160.795788 125.835203 \n",
       "L 181.787567 125.835203 \n",
       "L 181.903446 136.922727 \n",
       "L 231.900852 136.922727 \n",
       "L 231.900852 136.922727 \n",
       "\" clip-path=\"url(#p0616b0e99c)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 45.478125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 240.778125 143.1 \n",
       "L 240.778125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 45.478125 143.1 \n",
       "L 240.778125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 45.478125 7.2 \n",
       "L 240.778125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 159.996875 59.234375 \n",
       "L 233.778125 59.234375 \n",
       "Q 235.778125 59.234375 235.778125 57.234375 \n",
       "L 235.778125 14.2 \n",
       "Q 235.778125 12.2 233.778125 12.2 \n",
       "L 159.996875 12.2 \n",
       "Q 157.996875 12.2 157.996875 14.2 \n",
       "L 157.996875 57.234375 \n",
       "Q 157.996875 59.234375 159.996875 59.234375 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_91\">\n",
       "     <path d=\"M 161.996875 20.298438 \n",
       "L 171.996875 20.298438 \n",
       "L 181.996875 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- unigram -->\n",
       "     <g transform=\"translate(189.996875 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-75\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"63.378906\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"126.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-67\" x=\"154.541016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"218.017578\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"259.130859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"320.410156\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_92\">\n",
       "     <path d=\"M 161.996875 34.976562 \n",
       "L 171.996875 34.976562 \n",
       "L 181.996875 34.976562 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- bigram -->\n",
       "     <g transform=\"translate(189.996875 38.476562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-62\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-67\" x=\"91.259766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"154.736328\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"195.849609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"257.128906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_93\">\n",
       "     <path d=\"M 161.996875 49.654688 \n",
       "L 171.996875 49.654688 \n",
       "L 181.996875 49.654688 \n",
       "\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- trigram -->\n",
       "     <g transform=\"translate(189.996875 53.154688)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-67\" x=\"108.105469\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"171.582031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"212.695312\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"273.974609\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p0616b0e99c\">\n",
       "   <rect x=\"45.478125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bigram_freqs = [freq for token, freq in bigram_vocab.token_freqs]\n",
    "trigram_freqs = [freq for token, freq in trigram_vocab.token_freqs]\n",
    "d2l.plot([freqs, bigram_freqs, trigram_freqs], xlabel='token: x',\n",
    "         ylabel='frequency: n(x)', xscale='log', yscale='log',\n",
    "         legend=['unigram', 'bigram', 'trigram'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.203984Z",
     "iopub.status.busy": "2022-07-31T02:26:09.203477Z",
     "iopub.status.idle": "2022-07-31T02:26:09.210202Z",
     "shell.execute_reply": "2022-07-31T02:26:09.209574Z"
    },
    "origin_pos": 14,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def seq_data_iter_random(corpus, batch_size, num_steps):\n",
    "    \"\"\"使用随机抽样生成一个小批量子序列\"\"\"\n",
    "    corpus = corpus[random.randint(0, num_steps - 1):]\n",
    "    num_subseqs = (len(corpus) - 1) // num_steps\n",
    "    initial_indices = list(range(0, num_subseqs * num_steps, num_steps))\n",
    "    random.shuffle(initial_indices)\n",
    "\n",
    "    def data(pos):\n",
    "        return corpus[pos: pos + num_steps]\n",
    "\n",
    "    num_batches = num_subseqs // batch_size\n",
    "    for i in range(0, batch_size * num_batches, batch_size):\n",
    "        initial_indices_per_batch = initial_indices[i: i + batch_size]\n",
    "        X = [data(j) for j in initial_indices_per_batch]\n",
    "        Y = [data(j + 1) for j in initial_indices_per_batch]\n",
    "        yield torch.tensor(X), torch.tensor(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.213188Z",
     "iopub.status.busy": "2022-07-31T02:26:09.212749Z",
     "iopub.status.idle": "2022-07-31T02:26:09.218521Z",
     "shell.execute_reply": "2022-07-31T02:26:09.217850Z"
    },
    "origin_pos": 16,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:  tensor([[14, 15, 16, 17, 18],\n",
      "        [ 9, 10, 11, 12, 13]]) \n",
      "Y: tensor([[15, 16, 17, 18, 19],\n",
      "        [10, 11, 12, 13, 14]])\n",
      "X:  tensor([[19, 20, 21, 22, 23],\n",
      "        [29, 30, 31, 32, 33]]) \n",
      "Y: tensor([[20, 21, 22, 23, 24],\n",
      "        [30, 31, 32, 33, 34]])\n",
      "X:  tensor([[ 4,  5,  6,  7,  8],\n",
      "        [24, 25, 26, 27, 28]]) \n",
      "Y: tensor([[ 5,  6,  7,  8,  9],\n",
      "        [25, 26, 27, 28, 29]])\n"
     ]
    }
   ],
   "source": [
    "my_seq = list(range(35))\n",
    "for X, Y in seq_data_iter_random(my_seq, batch_size=2, num_steps=5):\n",
    "    print('X: ', X, '\\nY:', Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.221566Z",
     "iopub.status.busy": "2022-07-31T02:26:09.221038Z",
     "iopub.status.idle": "2022-07-31T02:26:09.226641Z",
     "shell.execute_reply": "2022-07-31T02:26:09.225987Z"
    },
    "origin_pos": 18,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def seq_data_iter_sequential(corpus, batch_size, num_steps):\n",
    "    \"\"\"使用顺序分区生成一个小批量子序列\"\"\"\n",
    "    offset = random.randint(0, num_steps)\n",
    "    num_tokens = ((len(corpus) - offset - 1) // batch_size) * batch_size\n",
    "    Xs = torch.tensor(corpus[offset: offset + num_tokens])\n",
    "    Ys = torch.tensor(corpus[offset + 1: offset + 1 + num_tokens])\n",
    "    Xs, Ys = Xs.reshape(batch_size, -1), Ys.reshape(batch_size, -1)\n",
    "    num_batches = Xs.shape[1] // num_steps\n",
    "    for i in range(0, num_steps * num_batches, num_steps):\n",
    "        X = Xs[:, i: i + num_steps]\n",
    "        Y = Ys[:, i: i + num_steps]\n",
    "        yield X, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.229678Z",
     "iopub.status.busy": "2022-07-31T02:26:09.229236Z",
     "iopub.status.idle": "2022-07-31T02:26:09.234468Z",
     "shell.execute_reply": "2022-07-31T02:26:09.233792Z"
    },
    "origin_pos": 21,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:  tensor([[ 4,  5,  6,  7,  8],\n",
      "        [19, 20, 21, 22, 23]]) \n",
      "Y: tensor([[ 5,  6,  7,  8,  9],\n",
      "        [20, 21, 22, 23, 24]])\n",
      "X:  tensor([[ 9, 10, 11, 12, 13],\n",
      "        [24, 25, 26, 27, 28]]) \n",
      "Y: tensor([[10, 11, 12, 13, 14],\n",
      "        [25, 26, 27, 28, 29]])\n",
      "X:  tensor([[14, 15, 16, 17, 18],\n",
      "        [29, 30, 31, 32, 33]]) \n",
      "Y: tensor([[15, 16, 17, 18, 19],\n",
      "        [30, 31, 32, 33, 34]])\n"
     ]
    }
   ],
   "source": [
    "for X, Y in seq_data_iter_sequential(my_seq, batch_size=2, num_steps=5):\n",
    "    print('X: ', X, '\\nY:', Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.237593Z",
     "iopub.status.busy": "2022-07-31T02:26:09.237033Z",
     "iopub.status.idle": "2022-07-31T02:26:09.241972Z",
     "shell.execute_reply": "2022-07-31T02:26:09.241324Z"
    },
    "origin_pos": 23,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "class SeqDataLoader:\n",
    "    \"\"\"加载序列数据的迭代器\"\"\"\n",
    "    def __init__(self, batch_size, num_steps, use_random_iter, max_tokens):\n",
    "        if use_random_iter:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_random\n",
    "        else:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_sequential\n",
    "        self.corpus, self.vocab = d2l.load_corpus_time_machine(max_tokens)\n",
    "        self.batch_size, self.num_steps = batch_size, num_steps\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self.data_iter_fn(self.corpus, self.batch_size, self.num_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 使用示例\n",
    "\n",
    "```python\n",
    "data_iter, vocab = load_data_time_machine(batch_size=32, num_steps=35)\n",
    "for X, Y in data_iter:\n",
    "    # 训练模型\n",
    "    pass\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-31T02:26:09.244926Z",
     "iopub.status.busy": "2022-07-31T02:26:09.244503Z",
     "iopub.status.idle": "2022-07-31T02:26:09.248402Z",
     "shell.execute_reply": "2022-07-31T02:26:09.247759Z"
    },
    "origin_pos": 25,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def load_data_time_machine(batch_size, num_steps,\n",
    "                           use_random_iter=False, max_tokens=10000):\n",
    "    \"\"\"返回时光机器数据集的迭代器和词表\"\"\"\n",
    "    data_iter = SeqDataLoader(\n",
    "        batch_size, num_steps, use_random_iter, max_tokens)\n",
    "    return data_iter, data_iter.vocab"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "rise": {
   "autolaunch": true,
   "enable_chalkboard": true,
   "overlay": "<div class='my-top-right'><img height=80px src='http://d2l.ai/_static/logo-with-text.png'/></div><div class='my-top-left'></div>",
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
